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ABSTRACT 

Multioperator  tasks  often  require  complex  cognitive  processing  at  the  team  level. 
Many  team  cognitive  processes  such  as  situation  assessment  and  coordination  are  thought 
to  rely  on  team  knowledge.  Team  knowledge  is  multifaceted  and  comprised  of  relatively 
generic  knowledge  in  the  form  of  team  mental  models  and  more  specific  team  situation 
models.  In  this  methodological  review  paper,  recent  efforts  to  measure  team  knowledge 
are  reviewed  in  the  context  of  mapping  specific  methods  onto  features  of  targeted  team 
knowledge.  Team  knowledge  features  include  type,  homogeneity  vs.  heterogeneity,  and 
rate  of  knowledge  change.  Measurement  features  include  knowledge  elicitation  method, 
team  metric,  and  aggregation  method.  When  available,  analytical  conclusions  or 
empirical  data  that  support  a  connection  between  team  knowledge  and  measurement 
method  are  highlighted.  Also,  empirical  results  concerning  the  relation  between  team 
knowledge  and  performance  are  presented  for  each  measurement  method  described  and 
research  and  methodological  needs  are  identified. 
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Measuring  Team  Knowledge 

Technological  developments  in  the  workplace  and  elsewhere  have  drastically 
changed  the  nature  of  many  tasks  (Howell  &  Cooke,  1989).  What  were  once  highly 
repetitive  tasks,  requiring  practiced  motor  skills,  are  now  tasks  that  require  cognitive 
skills  often  related  to  overseeing  new  technology  such  as  monitoring,  planning,  decision 
making,  and  design.  As  a  result,  a  full  understanding  of  many  tasks  requires  an 
examination  of  their  cognitive  underpinnings.  Taking  a  cognitive  engineering 
perspective,  these  cognitive  factors  need  to  be  examined  in  the  context  of  the  larger 
sociotechnical  system  in  which  they  are  embedded  (Hutchins,  1995;  Norman,  1986, 
Woods  &  Roth,  1988). 

For  instance,  the  growing  complexity  of  tasks  frequently  surpasses  the  cognitive 
capabilities  of  individuals  and  thus,  necessitates  a  team  approach.  Teams  play  an 
increasingly  critical  role  in  complex  military  operations  in  which  technological  and 
information  demands  necessitate  a  multioperator  environment  (Salas,  Cannon-Bowers, 
Church-Payne,  &  Smith- Jentsch,  1998).  In  addition,  civilian  applications  ranging  from 
manufacturing  to  nuclear  power  plant  operation  and  computer-supported  collaborative 
work  also  require  a  team  perspective  (e.g.,  Sundstrom,  DeMeuse,  &  Futrell,  1990). 

Salas,  Dickinson,  Converse,  and  Tannenbaum  (1992)  define  team  as  "a 
distinguishable  set  of  two  or  more  people  who  interact  dynamically,  interdependently, 
and  adaptively  toward  a  common  and  valued  goal/object/mission,  who  have  each  been 
assigned  specific  roles  or  functions  to  perfonn,  and  who  have  a  limited  life  span  of 
membership"  (p.  126-127).  Thus,  teams,  unlike  some  groups,  have  differentiated 
responsibilities  and  roles  (Cannon-Bowers,  Salas,  &  Converse,  1993).  This  division  of 
labor  enables  teams  to  tackle  tasks  too  complex  for  any  individual. 

Whereas  the  team  approach  is  often  seen  as  a  solution  to  cognitively  complex  tasks, 
it  also  introduces  an  additional  layer  of  cognitive  requirements  that  are  associated  with 
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the  demands  of  working  together  effectively  with  others.  Team  members  need  to 
coordinate  their  activities  with  others  who  are  working  toward  the  same  goal.  Team 
tasks  often  call  for  the  team  to  detect  and  recognize  pertinent  cues,  make  decisions,  solve 
problems,  remember  relevant  infonnation,  plan,  acquire  knowledge,  and  design  solutions 
or  products  as  an  integrated  unit.  Therefore,  an  understanding  of  team  cognition,  or 
what  some  have  called  the  new  "social  cognition"  (Klimoski  &  Mohammed,  1994),  is 
critical  to  understanding  much  team  performance. 

Parallel  to  research  on  individual  expertise  (e.g.,  Chase  &  Simon,  1973;  Glaser  & 
Chi,  1988),  accounts  of  effective  team  perfonnance  highlight  the  importance  of 
knowledge,  or  in  this  case  team  knowledge.  For  instance,  Cannon-Bowers  and  Salas 
(1997)  have  recently  proposed  a  framework  which  integrates  many  aspects  of  team 
cognition  in  the  form  of  teamwork  competencies.  They  categorize  competencies  required 
for  effective  teamwork  in  terms  of  knowledge,  skills,  and  attitudes  that  are  either  specific 
or  generic  to  the  task  and  specific  or  generic  to  the  team.  Similarly,  a  team's 
understanding  of  a  complex  and  dynamic  situation  at  any  one  point  in  time  (i.e.,  team 
situation  awareness)  is  supposedly  influenced  by  the  knowledge  that  the  team  possesses 
(Cooke,  Stout,  &  Salas,  1997;  Stout,  Cannon-Bowers,  &  Salas,  1996).  In  this  paper  we 
examine  the  measurement  of  team  knowledge,  describing  methods  and  metrics  that  have 
been  used,  proposing  additional  measures,  and  identifying  research  and  methodological 
needs  where  appropriate.  First,  however,  we  define  team  knowledge. 

Team  Knowledge:  A  Definition 

The  knowledge  possessed  by  effective  teams  has  been  frequently  referred  to  as 
shared  knowledge  and  in  similar  contexts,  as  shared  mental  models,  shared  cognition,  and 
shared  understanding  (Blickensderfer,  Cannon-Bowers,  &  Salas,  1997b;  Cannon- 
Bowers,  et  ah,  1993;  Converse,  Cannon-Bowers,  &  Salas,  1991;  Klimoski  & 

Mohammed,  1994;  Kraiger,  Krause,  &  Hart,  1996;  Kraiger  &  Wenzel,  1997;  Orasanu, 
1990;  Rentsch  &  Hall,  1994;  Robertson  &  Endsley,  1997;  Rouse,  Cannon-Bowers,  & 
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Salas,  1992;  Sarter  &  Woods,  1991;  Stout,  et  al.,  1996).  Such  knowledge  sharing  is 
thought  to  help  teams  coordinate  implicitly  when  explicit  communications  are  hampered, 
thereby  enhancing  team  performance  (Cannon-Bowers,  et  ah,  1993;  Kleinman  &  Serfaty, 
1989;  Stout,  et  ah,  1996).  For  example,  Cannon-Bowers,  et  al.,  (1993)  suggest  that 
shared  mental  models  provide  mutual  expectations  which  allow  teams  to  coordinate  and 
make  predictions  about  the  behavior  and  needs  of  their  teammates. 

To  date,  there  has  been  much  conceptual  work  on  shared  mental  models.  Notably, 
Cannon-Bowers,  et  al.  (1993),  Kraiger  and  Wenzel  (1997),  and  Klimoski  and 
Mohammed  (1994)  have  provided  extensive  reviews  of  work  in  this  area,  and  all  have 
attempted  to  explicate  the  relationship  between  shared  mental  models  and  team 
performance.  Others  have  tied  shared  mental  models  to  other  constructs  such  as  team 
situation  awareness  (Orasanu,  1990;  Robertson  &  Endsley,  1997;  Stout,  et.  al.,  1996; 
Wellens,  1993)  and  team  decision  making  (Walsh,  Henderson,  &  Deighton,  1988).  In 
addition,  there  are  many  factors  such  as  team  size,  task  type,  team  experience,  training, 
and  team  process  behaviors  that  are  hypothesized  to  affect  and  be  affected  by  shared 
mental  models  (Cannon-Bowers,  et  al.,  1993;  Klimoski  &  Mohammed,  1994;  Kraiger  & 
Wenzel,  1997).  Throughout  this  literature,  researchers  have  pointed  to  the  measurement 
of  shared  mental  models  as  the  missing  link  between  conceptualization  of  shared  mental 
models  and  further  research  on  the  construct.  Therefore,  we  take  this  conceptual  work 
on  shared  mental  models  as  a  point  of  departure  and  begin  to  identify  and  address  issues 
in  the  measurement  of  team  knowledge. 

As  a  first  step  in  this  process,  a  specific  definition  of  the  term  team  knowledge  is 
required.  We  prefer  the  term  team  knowledge  over  shared  mental  models  or  shared 
cognition  for  several  reasons.  First,  the  use  of  the  term  team,  as  described  previously, 
restricts  the  problem  domain  to  knowledge  possessed  by  teams,  as  opposed  to  dyads  or 
groups  in  general,  the  targets  for  related  work  such  as  common  ground  in  discourse 
(Clark,  1994)  and  group  information  processing  (Hinsz  &  Tindale,  1997).  Second,  we 
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prefer  to  avoid  use  of  the  term  shared  because,  as  we  discuss  later,  share  can  mean  to 
hold  in  common  or  to  apportion  and  the  ambiguity  can  create  confusion,  especially  in  this 
context.  Finally,  we  use  the  tenn  knowledge  rather  than  the  broader  term,  cognition 
because  the  latter  would  include  a  wide  range  of  cognitive  phenomena  at  the  team  level 
such  as  team  decision  making,  team  vigilance,  and  team  situation  awareness.  Whereas 
much  of  team  cognition  relies  heavily  on  team  knowledge,  the  former  is  beyond  the 
scope  of  this  review.  Thus,  this  paper  focuses  on  team  knowledge  which  is  a  subset  of 
team  cognition  as  represented  in  the  framework  of  Table  1.  Furthermore,  we  prefer  the 
term  knowledge  over  the  narrower  term,  mental  model  in  order  to  capture  the  two  facets 
of  team  knowledge  also  illustrated  in  Table  1. 

[Insert  Table  1  about  here] 

In  their  synthesis  of  the  literature,  Cannon-Bowers,  Salas,  and  Blickensderfer 
(1999)  distinguish  two  types  of  team  knowledge.  Team  mental  models  refer  to  the 
collective  task-  and  team-relevant  knowledge  that  team  members  bring  to  a  situation. 
Such  knowledge  is  acquired  by  team  members  through  formal  training,  experience,  team 
discussions,  and  the  like  and  is  relatively  long  lasting. 

The  knowledge  associated  with  the  team  mental  model  may  be  declarative  (i.e.,  the 
facts,  figures,  rules,  relations  and  concepts  in  a  task  domain),  procedural  (i.e.,  the  steps, 
procedures,  sequences,  and  actions  required  for  task  performance),  or  strategic  (the 
overriding  task  strategies  and  knowledge  of  when  they  apply)  (see  Stout,  et  al.,  (1996) 
for  further  explanation).  Its  content  can  include  knowledge  relevant  to  teamwork  such  as 
knowledge  of  team  member  roles  and  responsibilities  and  knowledge  of  teammates' 
knowledge,  skills,  abilities,  beleifs,  preferences,  and  style,  as  well  as  knowledge  relevant 
to  taskwork  such  as  cue-strategy  associations,  understanding  of  task  procedures,  and 
knowledge  of  typical  task  strategies  (see  Cannon-Bowers  ,  Tannenbaum,  Salas,  and 
Volpe  (1995)  for  more  on  required  knowledge  in  teams).  The  team  mental  model 
provides  a  collective  knowledge  base  for  team  members  to  draw  upon  when  task  episodes 
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ensue.  We  assume  that  the  content  of  the  team  mental  model  is  critically  related  to 
effective  team  performance. 

In  addition  to  team  mental  models,  another  type  of  team  knowledge,  the  team 
situation  model,  develops  in  situ  while  the  team  is  actually  engaged  in  the  task  (Orasanu, 
1990).  At  the  individual  level  team  members  acquire  a  specific  understanding  of  the 
current  situation  at  any  one  point  in  time  (i.e.,  a  situation  model).  This  understanding 
should  change  with  changes  in  the  situation  and  thus,  Cannon-Bowers,  et  ah,  (1999)  refer 
to  it  as  dynamic  understanding.  The  team  situation  model  is  the  team’s  collective 
understanding  of  the  specific  situation. 

According  to  Cannon-Bowers,  et  al.  (1999),  this  type  of  team  knowledge  is 
qualitatively  different  from  that  associated  with  the  team  mental  model,  in  that  it  makes 
use  of  the  pre-existing  team  mental  model,  but  goes  further  by  incorporating  the  specific 
characteristics  of  the  current  situation.  For  example,  during  task  performance,  team 
members  interpret  cues  and  patterns  in  a  situation  (Stout,  Salas,  &  Cannon-Bowers,  in 
press).  The  collective  outcome  of  these  interpretations  is  influenced  by  team  mental 
models  and  by  processes  employed  by  team  members  (e.g.,  the  clarity  of  team 
communications)  (Stout  et  al.,  1996). 

The  team  situation  model  guides  the  team  in  assessing  additional  cues  and  patterns 
in  the  situation,  determining  strategies  available  to  the  team,  assessing  how  the  team  is 
proceeding,  predicting  what  teammates  will  do  and  need,  and  selecting  appropriate 
actions  to  take.  Thus,  the  degree  to  which  teammates  are  coordinated  in  these  judgments 
is  crucial  to  team  performance  and  depends  on  the  team  situation  model. 

In  sum,  the  term  knowledge  is  meant  to  extend  beyond  mental  models  of  a  system, 
to  include  knowledge  about  other  task  and  team-relevant  areas,  as  well  as  the  more 
fleeting  interpretation  of  the  current  situation.  The  use  of  the  term  knowledge  is  also 
meant  to  be  neutral  as  to  the  accuracy  or  completeness  of  that  infonnation,  a  dimension 
which  can  however,  be  assessed  with  access  to  a  definitive  standard  as  discussed  later. 
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In  addition,  we  use  the  term  knowledge  pragmatically,  assuming  little  about  its  form  of 
representation  (i.e.,  connectionist  networks,  images,  symbolic  networks)  or  even  the 
importance  of  representation.  Instead  we  assume  that  teams,  like  individuals,  possess 
knowledge  and  that  this  knowledge  is  reflected  in  actions  or  behaviors  that  provide  us 
with  approximations  of  that  knowledge  that  can  be  applied  to  training  and  design. 

Thus,  team  knowledge  can  be  defined  as  the  collection  of  task-  and  team-related 
knowledge  held  by  teammates  and  their  collective  understanding  of  the  current  situation. 
Team  performance  will  be  maximized  to  the  extent  that  team  knowledge  is  accurate, 
appropriately  apportioned  among  members,  and  structured  in  such  a  way  as  to  support 
compatible  assessments  of  the  task  situation  and  development  of  effective  team  strategies 
to  cope  with  it  (Cannon-Bowers,  et  al.,  1999). 

Measuring  Team  Knowledge:  A  Methodological  Review 

Although  team  knowledge  and  related  constructs  have  been  assumed  to  be  critical 
aspects  of  team  performance,  there  has  been  minimal  empirical  work  to  support  this 
claim.  Research  has  been  focused  on  the  conceptualization  of  the  construct  and  empirical 
investigations  await  the  development  and  evaluation  of  measures  of  team  knowledge. 
Measures  of  team  knowledge  are  not  only  needed  to  better  understand  the  underpinnings 
of  effective  team  performance,  but  in  addition,  they  are  central  to  knowledge-intensive 
applications  such  as  team  training  programs  or  the  design  of  knowledge-aiding 
technologies  (e.g.,  Neerincx  &  deGreef,  1998,  Stout,  et  al.,  1996).  The  content  and 
structure  of  team  knowledge  associated  with  effective  team  perfonnance  can  feed  directly 
into  the  content  of  training  materials.  In  addition,  such  infonnation  can  be  used  to  assess 
the  current  state  of  a  team's  knowledge  in  order  to  identify  training  needs  or  evaluate  the 
success  of  training  interventions.  In  the  remainder  of  this  paper  we  review 
methodological  research  directed  at  the  measurement  of  team  knowledge  in  the  context 
of  the  goal  of  mapping  the  characteristics  of  the  targeted  team  knowledge  onto  the 
specific  measurement  approach. 
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Measures  can  only  be  evaluated  in  the  context  of  the  measurement  target.  For 
instance,  the  average  height  of  a  team’s  members  is  not  alone  a  valid  or  invalid  measure 
unless  a  target  such  as  team  basketball  skill  or  team  intelligence  is  revealed.  Similarly, 
measures  of  team  knowledge  should  be  developed,  applied,  and  evaluated  in  the  context 
of  the  targeted  team  knowledge.  This  is  because  team  knowledge  is  multifaceted  and 
different  measures  will  yield  different  information  about  team  knowledge.  In  the 
context  of  knowledge  elicitation  methods  for  individuals,  the  proposal  that  different 
methods  elicit  different  types  of  knowledge,  has  been  labeled  the  differential  access 
hypothesis  (Hoffman,  Shadbolt,  Burton,  &  Klein,  1995).  In  fact,  the  methods  can  be 
viewed  as  tools  in  a  tool  kit  (or  paints  on  a  palette  (Hoffman,  et  ah,  1995))  and  one  would 
not  select  a  hammer  to  tighten  a  screw.  Congruent  with  this  analogy,  it  is  likely  that 
multiple  "tools"  are  required  to  measure  a  specific  aspect  of  team  knowledge  (Hinsz, 
1995;  Kraiger  &  Wenzel,  1997). 

Thus,  our  review  of  team  knowledge  measures  focuses  on  both  the  targeted 
characteristics  of  team  knowledge,  as  well  as  the  characteristics  of  the  measurement 
approach.  Figure  1  provides  an  advanced  look  at  the  characteristics  of  each  which  are 
covered  in  detail  in  our  review.  Note  that  these  characteristics  are  not  meant  to 
completely  capture  team  knowledge  or  team  knowledge  measures;  yet  they  are 
representative  of  our  current  thinking,  as  well  as  that  of  the  recent  literature.  Ideally,  as 
also  represented  in  Figure  1,  the  characteristics  of  the  targeted  knowledge  should  map 
directly  onto  a  specific  measurement  approach.  Mappings  suggested  by  the  sparse 
literature  in  this  area  are  highlighted  in  this  review.  Other  unaddressed  mappings  can 
best  be  viewed  as  a  guide  for  additional  research  in  this  relatively  new  area.  The  review 
begins  with  identification  of  the  features  of  team  knowledge  that  should  be  considered  in 
selecting  a  measurement  approach. 


[Insert  Figure  1  about  here] 
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Characteristics  of  Team  Knowledge 

In  general,  the  type  of  team  knowledge  that  is  targeted  depends  on  the  subject 
domain  and  the  purpose  of  the  measurement.  For  instance,  in  order  to  assess  a  team’s 
situation  awareness  for  training  purposes  in  a  highly  dynamic  cockpit  environment,  one 
may  target  situation  models  However,  if  measurement  were  done  in  the  context  of  a 
design  group  with  the  purpose  of  facilitating  the  team  design  process,  then  mental  models 
of  the  task  may  be  targeted. 

In  Figure  1,  three  dimensions  along  which  team  knowledge  varies  are  represented: 
1)  mental  models  vs.  situation  models,  2)  homogeneous  vs.  heterogeneous  knowledge 
distribution,  and  3)  rate  of  change.  The  distinction  between  team  mental  models  and 
team  situation  models  was  discussed  in  the  previous  section.  Further,  this  distinction  can 
be  extended  by  including  more  specific  characterizations  of  knowledge  such  as  the  form 
(i.e.,  declarative,  procedural,  strategic)  and  the  content  (i.e.,  taskwork,  teamwork)  of  the 
team  knowledge.  The  second  and  third  distinctions  are  described  in  the  sections  that 
follow. 

Heterogeneity  and  Knowledge  Distribution 

It  has  been  suggested  that  the  view  that  effective  teams  have  shared  knowledge  or 
shared  mental  models  in  the  sense  of  common  or  similar  knowledge  is  overly  simplistic. 
Instead,  team  members  may  hold  in  addition  to  common  knowledge,  compatible  or 
complimentary  knowledge  (Cannon-Bowers  &  Salas,  1997;  Klimoski  &  Mohammed, 
1994).  That  is,  there  may  be  some  knowledge  overlap  required  among  team  members, 
but  in  addition  role-specific,  yet  compatible  knowledge  is  required.  Such  team 
heterogeneity,  in  which  different  team  members  are  assigned  different  roles,  is  featured 
in  the  well-accepted  definition  of  teams  offered  earlier  and  seems  characteristic  of  some 
of  the  most  interesting  team  tasks.  Consider,  for  example,  a  surgical  team.  In  some 
instances,  the  nurse  and  the  surgeon  may  need  to  have  some  knowledge  that  is  held  in 
common.  However,  the  nurse  is  not  likely  to  be  able  to  understand,  or  need  to 
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understand,  all  of  the  surgeon’s  knowledge;  hence,  he  or  she  must  have  some  knowledge 
that  is  compatible  with  the  surgeon's  knowledge,  but  not  necessarily  identical. 

Cases  in  which  team  knowledge  is  identical  or  completely  distinct  are  highly 
unlikely  (Rentsch  &  Hall,  1994).  In  fact,  common  team  knowledge  in  its  extreme  would 
seem  ripe  for  group  think  (Janis,  1972)  and  according  to  Klimoski  and  Mohammed  (1994) 
"...completely  overlapping  team  mental  models  are  viewed  as  dysfunctional  with  regard 
to  team  performance"  (p.  420).  That  is,  tasks  that  require  teams  are  likely  to  be  tasks 
which  are  so  complex  that  they  need  to  be  divided  among  several  individuals  who  are 
responsible  for  distinct  subtasks.  Cannon-Bowers,  et  al.  (1993)  also  take  the  view  that 
whereas  a  certain  degree  of  overlap  among  mental  models  of  team  members  is  needed  for 
effective  coordination  and  shared  expectations,  there  is  also  probably  a  point  at  which 
too  much  overlap  will  result  in  group  think  or  become  a  liability.  For  instance,  in  the 
domain  of  group  decision  making,  Walsh  et  al.,  (1988)  assume  that  knowledge  similarity 
is  undesirable. 

One  source  of  confusion  in  regard  to  this  issue  and  alluded  to  earlier  is  that  the  tenn 
shared  has  multiple  senses  (Klimoski  &  Mohammed,  1994).  Share  can  mean  to  have  in 
common  (as  in  share  the  equipment  or  share  the  belief)  or  it  can  mean  to  divide  (e.g., 
share  the  workload  or  share  the  dessert).  Likewise,  shared  knowledge  may  refer  to 
either  knowledge  that  is  similar  within  a  team  (i.e.,  homogeneous  with  respect  to  team 
members)  or  knowledge  that  is  distributed  among  team  members  (i.e.,  heterogeneous). 
For  most  tasks,  both  situations  are  likely  to  exist .  Therefore,  whereas  the  dichotomy 
represented  in  Figure  1  may  be  appropriate  for  describing  specific  units  of  team 
knowledge,  a  point  on  a  continuum  ranging  from  homogeneous  to  heterogeneous  is 
probably  more  descriptive  of  team  knowledge  as  a  whole. 

Assuming  some  degree  of  knowledge  heterogeneity  on  the  team,  it  is  necessary  to 
understand  the  distributional  requirements  of  the  knowledge  base  that  are  associated  with 
effective  team  performance.  Each  team  member  fulfills  an  independent  role  on  the  team, 
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as  well  as  an  interdependent  role  as  team  member.  Therefore,  measuring  team 
knowledge  is  not  simply  a  question  of  measuring  everything  each  team  member  knows. 
Instead,  it  is  necessary  to  discriminate  between  knowledge  that  is  unique  to  each  team 
member  and  required  by  other  team  members,  as  opposed  to  unique  knowledge  which  is 
only  required  by  the  individual  in  that  role.  Thus,  the  measurement  target  is  not  all  that  a 
team  knows,  but  within  the  confines  of  a  specific  task  and  environment,  the  knowledge 
required  by  interdependent  team  members  and  its  distributional  characteristics  across 
those  team  members. 

Rate  of  Change 

Rate  of  change  indicated  in  Figure  1  has  to  do  with  the  speed  with  which  the  team 
knowledge  changes,  with  slow,  average,  and  fast  being  arbitrary  markers  on  a  continuous 
scale.  Change  is  a  integral  feature  of  team  knowledge.  Team  knowledge  often  exists 
within  the  context  of  a  dynamic  environment.  Thus,  the  team  situation  model  is  in  a 
constant  state  of  flux. 

In  addition,  the  longer-lasting  team  mental  models,  like  individual  mental  models, 
evolve  with  experience  and  knowledge  acquisition.  However,  in  the  team  domain, 
changes  can  be  expected  to  occur  much  more  rapidly  than  at  the  individual  level,  because 
any  individual  change  can  potentially  change  the  overall  team  knowledge.  Change  is 
likely  to  be  even  more  exaggerated  in  experiments  in  which  college  undergraduates  with 
little  or  no  task  experience  serve  as  team  members.  In  short,  measuring  team  knowledge 
under  these  dynamic  conditions  is  tantamount  to  aiming  at  a  moving  target. 

As  an  initial  solution,  investigators  have  opted  for  knowledge  measures  taken  at 
several  discrete  points  during  an  experimental  session  (e.g.,  Mathieu,  Heffner,  Goodwin, 
Salas,  &  Cannon-Bowers,  in  press),  however,  repetitive  interruptions  of  a  team  for 
elicitation  during  a  task  can  also  be  disruptive  and  the  result  is  more  like  a  series  of  static 
snapshots,  rather  than  a  continuous  video  record.  Given,  the  predominance  of  task 
domains  at  the  fast  end  of  the  rate  of  change  scale,  more  attention  needs  to  be  directed  at 
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data  collected  continuously  and  synchronously  with  task  performance  (i.e.,  video 
records,  computer  event  records,  communication  records,  performance  measures). 

The  characterization  of  targeted  team  knowledge  in  terms  of  type,  distribution,  and 
rate  of  change,  paves  the  way  for  the  selection  of  a  measurement  approach.  In  the 
following  sections  features  that  differentiate  various  approaches  are  described. 

Characteristics  of  Measurement  Approaches 

Measurement  approaches  can  be  characterized  by  the  three  features  that  are 
represented  in  Figure  1(1)  elicitation  method,  2)  team  metric,  and  3)  aggregation 
method)  and  described  in  detail  in  the  sections  that  follow.  These  features  together 
comprise  what  might  be  called  a  collective  approach  to  measuring  team  knowledge.  That 
is,  team  knowledge  is  viewed  as  a  collection  of  the  knowledge  of  the  individual  team 
members.  This  approach  is  representative  of  the  research  to  date  on  the  measurement  of 
team  knowledge.  However,  before  describing  the  features  of  the  collective  approach  we 
introduce  an  alternative  approach  which  might  be  labeled  a  holistic  approach  to  team 
knowledge  measurement. 

Klimoski  and  Mohammed  (1994)  point  out  that  team  knowledge  is  more  than  the 
collection  of  knowledge  of  individual  team  members,  but  instead  emerges  as  a  result  of 
interactions  among  team  members.  In  fact,  Hutchins  (1991)  demonstrates  through 
simulation  that  team  knowledge  can  be  more  than  knowledge  external  or  internal  to  team 
members,  but  is  the  consequence  of  interactions  of  factors  in  the  sociotechnical  system. 
This  concept  of  emergent  knowledge  in  the  context  of  the  collective  and  holistic 
approaches  is  illustrated  in  Figure  2.  Whereas  the  collective  approach  targets  the 
knowledge  of  individual  team  members  and  then  aggregates  this  information,  the  holistic 
approach  targets  the  team  knowledge  that  results  from  the  application  of  team  process 
behaviors  (i.e.,  communication,  situation  assessment,  coordination)  to  the  collective 
knowledge.  Thus,  the  holistic  target  mediates  between  team  process  behaviors  and  team 
performance.  It  can  be  conceptualized  as  effective  team  knowledge  or  the  knowledge 
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that  is  revealed  in  the  team's  actions.  Whereas  both  collective  and  holistic  targets  should 
ultimately  be  reflected  in  team  performance,  the  holistic  target  is  more  directly  relevant. 
In  short,  the  result  of  the  collective  approach  underestimates  the  importance  of  team 
process  behaviors  in  determining  what  the  team  knows,  and  simply  assumes  that  team 
knowledge  is  equivalent  to  the  sum  of  what  each  team  member  knows. 

[Insert  Figure  2  about  here.] 

For  example,  in  the  case  of  a  two-person  cockpit  crew,  the  pilot  may  know  of 
impending  threat  at  point  A  and  the  co-pilot  knows  of  a  threat  at  point  B.  The 
aggregation  of  these  two  facts  represent  the  outcome  of  the  collective  approach. 

However,  the  outcome  of  the  holistic  approach  might  differ.  In  the  case  of  poor  team 
process  (e.g.,  perhaps  silence  on  the  co-pilot's  part  due  to  authority  issues),  effective  team 
knowledge  may  be  less  than  this  (knowledge  of  threat  at  point  A  only).  This  could  be 
reflected  in  the  team's  decision  to  progress  to  point  B.  Alternatively,  more  effective  team 
process  may  lead  the  team  to  put  their  infonnation  together,  along  with  their  background 
knowledge,  to  infer  that  there  is  also  a  probable  threat  at  point  C.  Again,  such  knowledge 
would  be  reflected  in  their  behavior  and  ultimate  selection  of  a  destination. 

A  holistic  approach  to  knowledge  measurement  would  require  new  methods, 
perhaps  interviewing  the  team  as  a  whole,  for  instance.  The  holistic  approach  also  raises 
a  host  of  interesting  questions  such  as  the  impact  of  social  or  political  roles  on  the 
outcome  of  the  elicitation.  Research  is  clearly  needed  to  comparatively  evaluate 
collective  and  new  holistic  approaches.  This  said,  the  measurement  characteristics 
discussed  in  this  section  pertain  to  the  collective  approach  in  that  elicitation  of  team 
knowledge  is  conducted  at  the  individual  level  and  aggregated  across  team  members  to 
reflect  team  knowledge. 

Knowledge  Elicitation  Method 

Team  knowledge  measurement,  like  individual  knowledge  measurement,  requires 
methods  that  go  beyond  assessment  of  knowledge  accuracy  to  capture  the  richness  of  the 
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knowledge  itself.  Two  individuals  may  perform  at  similar  levels  of  accuracy,  but  have 
widely  different  views  of  the  situation.  Thus,  meaningful  judgments  about  team 
knowledge  are  impossible  to  make  on  the  basis  of  accuracy  alone.  The  measure  must 
include  information  on  the  content  and  structure  of  the  team’s  knowledge.  Knowledge 
elicitation  methods  have  been  used  to  satisfy  these  goals  at  the  individual  level  and 
recently  investigators  have  begun  to  apply  these  methods  to  the  team  level  (e.g., 

Gualtieri,  Fowlkes,  &  Ricci,  1996;  Kraiger,  et  ah,  1996;  Mathieu,  et  al.,  in  press;  Stout, 
1995;  Minionis,  Zaccaro,  &  Perez,  1995;  Walsh,  et  al.,  1988). 

At  the  individual  level,  knowledge  elicitation  is  the  practice  of  explicating  the 
domain-related  knowledge,  often  mental  model-like  knowledge,  held  by  an  individual. 

In  many  cases  the  individual  is  an  expert  in  the  domain  and  the  elicitation  is  conducted 
with  the  goal  of  better  understanding  the  domain.  More  detailed  definitions,  along  with 
historical  development  of  the  enterprise  and  associated  issues  can  be  found  in  Cooke 
(1994),  Cooke  (1999),  and  Hoffman,  et  al.  (1995). 

A  wide  variety  of  knowledge  elicitation  methods  has  been  applied  to  the  elicitation 
of  knowledge  at  the  individual  level.  Most  of  these  methods  fall  into  one  of  the  four 
categories  listed  in  Figure  1  :  1)  observations,  2)  interviews  and  surveys,  3)  process 
tracing,  and  4)  conceptual  methods.  Each  category  contains  a  number  of  methods  that 
vary  in  terms  of  the  specific  procedure  and  the  type  of  knowledge  that  is  targeted. 
Whereas  details  can  be  found  in  the  aforementioned  reviews,  a  brief  overview  of  each 
category  and  its  relevance  to  team  knowledge  measurement  follows. 

Methods  for  eliciting  knowledge.  Observations  can  take  written,  audio,  or  video 
forms  and  have  the  potential  to  provide  vast  amounts  of  infonnation  in  the  context  of 
performance,  while  interfering  minimally  with  tasks.  Thus,  observations  of  team 
behavior  provide  a  continuous  source  of  data  that  reflects  team  knowledge.  These  data 
could  potentially  serve  as  an  alternative  to  static  snapshots  of  team  knowledge  in  cases  in 
which  rapid  change  is  indicated.  Variations  within  the  category  of  observational  methods 
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revolve  around  issues  of  how  to  observe  (passively  or  actively)  and  what  to  observe 
(predetennined  features  or  specific  aspects  of  the  task  and  environment).  The  advantage 
of  minimal  intrusion  is  offset  by  difficulties  interpreting  the  data.  Recently,  however, 
tools  to  aid  in  this  process  such  as  video  analysis  software  have  been  developed  (Harrison 
&  Baecker,  1991). 

Interviews  and  surveys,  like  observations,  are  excellent  for  gaining  a  general 
understanding  of  the  situation,  and  generating  and  verifying  hypotheses.  Unstructured 
interviews  are  free-fonn  in  that  neither  content  nor  sequencing  is  prespecified. 

Structured  interviews  follow  a  predetermined  format  and  can  vary  from  highly  rigid  to 
only  loosely  constrained.  Although  structured  interviews  require  more  preparation  time 
than  unstructured  interviews,  they  also  have  advantages  of  being  more  systematic,  and 
therefore  more  complete  and  comfortable  for  both  participants.  With  enough  structure,  a 
structured  interview  can  be  converted  into  a  paper-and-pencil  measure,  such  as  a  survey 
or  questionnaire  which  are  often  easier  to  administer  than  interviews.  In  fact,  some 
investigators  have  used  structured  interviews  in  the  form  of  written  questionnaires  to 
elicit  team  knowledge  (Blickensderfer,  et  al.,  1997b;  Blickensderfer,  Cannon-Bowers,  & 
Salas,  1997a;  Jenkins  &  Rentsch,  1995).  Relative  to  other  elicitation  methods,  interviews 
and  questionnaires  have  been  used  extensively  to  measure  team  knowledge.  In 
particular,  because  of  their  independence  from  task  performance,  they  have  been  used 
chiefly  tomeasure  team  mental  models  which  are  thought  to  be  less  context-dependent 
and  more  stable  over  time  than  team  situation  models. 

Process  tracing  techniques  are  methods  for  collecting  data  concurrently  with  task 
performance.  These  data  are  later  analyzed  to  make  inferences  about  the  knowledge  or 
underlying  task  performance.  One  of  the  most  popular  forms  of  process  tracing  relies  on 
verbal  reports  provided  while  the  participant  is  "thinking  aloud"  during  task  perfonnance 
(vanSomeren,  Barnard,  &  Sandberg,  1994).  These  reports  can  be  retrospective,  but  it  is 
typically  believed  that  as  long  as  there  is  no  interference  with  the  task  itself,  concurrent 
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reports  provide  more  valid  information  regarding  the  current  contents  of  working 
memory.  Nonverbal  data  including  keystrokes,  actions,  facial  expressions,  gestures, 
and  general  behavioral  events  have  also  been  collected  to  trace  cognitive  processes  and 
reveal  underlying  procedural  knowledge.  Like  observations,  process  tracing  data  because 
of  its  on-line  nature,  lends  itself  well  to  the  elicitation  of  quickly  changing  team 
knowledge. 

Protocol  analysis  is  the  term  used  to  describe  various  methods  for  summarizing  and 
interpreting  process  tracing  data  (Ericsson  &  Simon,  1996).  It  typically  involves 
transcribing  the  data,  developing  a  coding  scheme  that  captures  the  critical  content  of  the 
data,  applying  the  coding  scheme  to  each  identified  unit  in  the  protocol,  and  exploring 
frequencies,  patterns,  and  sequential  dependencies  in  the  results.  A  relatively  new  area 
of  inquiry  that  focuses  on  ESDA  (Exploratory  Sequential  Data  Analysis;  Sanderson  & 
Fisher,  1994)  has  produced  some  analytical  tools  and  methods  in  recent  years  to  facilitate 
this  process.  The  combination  of  these  new  tools  and  event  data  captured  on-line  has 
promise  as  a  way  of  overcoming  many  of  the  costs  associated  with  knowledge  elicitation, 
as  well  as  providing  alternative  to  interviews  in  rapidly  changing  environments. 

Conceptual  methods  produce  representations  of  domain  concepts  and  their 
relations.  Methods  included  in  this  set  are  cluster  analysis  (Johnson,  1967), 
multidimensional  scaling  (Shepard,  1962a;  1962b),  Pathfinder  (Schvane veldt,  1990; 
Schvaneveldt,  Durso,  &  Dearholt,  1989),  and  concept  mapping  (Jonassen,  Beissner,  & 
Yacci,  1993).  In  general,  the  methods  take  pairwise  estimates  of  relatedness  for  a  set  of 
concepts  and  generate  a  spatial  or  graphical  representation  of  those  concepts  and  their 
relations.  The  general  goal  of  these  methods  is  to  reduce  the  set  of  distance  estimates  in  a 
meaningful  way.  Resulting  representations  can  then  be  compared  qualitatively  and  in 
many  cases,  quantitatively  across  groups  and  individuals.  The  techniques  tend  to  be 
indirect  in  that  they  require  judgments  about  conceptual  relatedness,  as  opposed  to 
introspections  or  explicit  verbal  reports.  One  advantage  of  these  methods,  particularly 
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for  measuring  team  knowledge,  is  that  they  can  handle  results  from  multiple  individuals, 
including  summarizing  data  from  several  individuals  and  comparing  data  across 
individuals  or  groups.  For  this  pragmatic  reason,  and  because  these  methods  target 
knowledge  structure  which  is  thought  to  be  associated  with  team  performance  (e.g., 
Cannon-Bowers  et  ah,  1993;  Klimoski  &  Mohammed,  1994),  applications  of  knowledge 
elicitation  methods  to  team  knowledge  not  using  interviews  and  surveys,  have  relied  on 
conceptual  methods  (Blickensderfer,  et  ah,  1997b;  Gualtieri,  et  ah,  1996;  Kraiger,  et  al. 
1996;  Mathieu,  et  al.,  in  press;  Stout,  1995;  Minionis,  et  al.,  1995;  Walsh,  et  al.,  1988). 

These  four  categories  represent  the  most  commonly  used  methods  for  eliciting 
knowledge  at  the  individual  level.  However,  they  are  not  all-inclusive.  Task  analysis 
methods,  which  analyze  a  task  into  it  components  (Kirwan  &  Ainsworth,  1992)  and 
policy  capturing  approaches,  which  quantify  decision  policies  (Dawes,  1979),  target  very 
specific  aspects  of  individual  knowledge  and  likewise,  are  additional  possibilities  for 
measuring  team  knowledge. 

Mapping  elicitation  methods  to  team  knowledge.  There  has  been  some  analytical 
and  empirical  work  directed  at  mapping  methods  for  individual  knowledge  elicitation  to 
type  of  knowledge  that  each  elicits  (Boose  &  Bradshaw,  1987;  Geiwitz,  Klatsky,  & 
McCloskey,  1988;  Geiwitz,  Kornell,  &  McCloskey,  1990;  Kitto  &  Boose,  1987,  1989; 
Meyer  &  Booker,  1990;  Wielinga,  Schreiber,  &  Breuker,  1992).  From  this  information 
and  the  team  knowledge  measurement  work  done  to  date  we  can  derive  some  preliminary 
conclusions  regarding  the  mapping  of  team  knowledge  to  elicitation  method. 

First,  most  of  the  approaches  used  thus  far  to  elicit  team  knowledge  have  focused 
on  the  elicitation  of  team  mental  models.  Specifically,  interviews  and  paper-and-pencil 
questionnaires,  along  with  conceptual  methods  such  as  multidimensional  scaling  and 
concept  maps,  are  best  suited  for  eliciting  knowledge  that  is  more  stable  and  less 
situation-dependent  than  that  associated  with  team  situation  models.  Instead,  these 
methods  are  best-suited  for  eliciting  team  mental  models  in  that  they  are  typically 
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administered  off-line  and  are  somewhat  divorced  from  the  immediate  task  context.  This 
is  not  to  say  that  team  mental  models  are  context-free,  but  rather  they  represent  a  more 
generic  understanding  of  the  task,  the  team,  or  the  environment  that  transcends  a  specific 
situation.  Also  we  can  assume  that  the  team  mental  model  evolves  over  time  with 
experience  and  knowledge  acquisition  of  the  team  members,  but  on  a  time  scale  of  weeks 
or  months,  so  that  in  the  time  that  it  takes  to  administer  interviews,  questionnaires,  and 
conceptual  methods,  the  change  has  not  been  dramatic. 

This  is  not  the  case  for  the  ever-changing  team  situation  model.  The  team  situation 
model  changes  in  concert  with  changes  in  the  situation,  which  for  many  complex  team 
tasks  occurs  at  a  rapid  clip— too  rapid  for  the  off-line  elicitation  methods  to  capture. 
Further,  because  the  team  situation  model  is  knowledge  that  represents  the  teams  current 
understanding  of  the  situation,  it  is  extremely  tied  to  the  context  of  that  situation.  Off¬ 
line  elicitation  methods  such  as  interviews  and  conceptual  methods  are  generally  weak  at 
capturing  the  nuances  and  the  information  content  of  a  specific  situation.  The  on-line 
methods  of  observations  and  process  tracing  appear  more  promising  for  rapidly  changing 
team  knowledge  and  specifically,  for  team  situation  models. 

Conceptual  methods  can  also  be  adapted  for  the  on-line  elicitation  required  for 
situation  model  elicitation  and  in  general,  for  elicitation  in  rapidly  changing  knowledge 
environments.  For  instance,  the  Pathfinder  method  described  previously  has  been 
applied  to  the  representation  of  sequential  behavioral  data  using  a  methodology  called 
PRONET  (Cooke,  Neville,  &  Rowe,  1996).  Another  way  to  adapt  off-line  knowledge 
elicitation  methods  to  team  situation  model  measurement  is  to  stream-line  them.  In  this 
case  the  goal  is  to  make  data  collection  (and  even  analysis  and  interpretation)  stages 
more  systematic  and  efficient.  For  instance,  one  way  to  streamline  the  pairwise  rating 
method  is  to  identify  through  prior  testing,  a  subset  of  concept  pairs  for  which  rating 
responses  were  particularly  diagnostic  of  team  situation  models.  Ratings  would  then  be 
collected  on  this  subset  as  opposed  to  all  pairs. 
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For  the  most  part,  however,  elicitation  methods  have  not  been  applied  to  team 
situation  model  measurement.  Query  methods  such  as  SAGAT  (Situation  Awareness 
Global  Assessment  Technique;  Endsley,  1990)  and  SPAM  (Situation  Present  Assessment 
Method;  Durso,  Hackworth,  Truitt,  Crutchfield,  Nikolic,  &  Manning,  1996)  have  been 
used  to  capture  an  individual's  fleeting  understanding  of  a  situation  and  there  have  been 
some  proposals  for  using  these  methods  to  assess  team  situation  awareness  as  well 
(Cooke,  et  ah,  1997).  However,  these  are  typically  very  specific  queries  such  as  "what  is 
your  current  heading,"  and  are  better  thought  of  as  assessment  methods,  rather  than 
elicitation  methods,  though  elicitation  (usually  in  the  form  of  interviews)  is  needed  to 
identify  diagnostic  queries. 

In  sum,  a  subset  of  knowledge  elicitation  methods  has  been  applied  to  limited 
varieties  of  team  knowledge.  The  measurement  of  situation  models  and  knowledge  in 
more  dynamic  domains  has  received  relatively  little  attention.  The  application  of  a 
broader  spectrum  of  knowledge  elicitation  methods  to  the  problem  of  measuring  team 
knowledge  should  open  the  door  to  the  measurement  of  varieties  of  team  knowledge 
previously  untapped. 

Team  metrics 

Although  knowledge  elicitation  methods  produce  graphical  structures  or  prototcols 
that  provide  qualitative  data  about  team  knowledge,  for  many  reasons  it  is  desirable  to 
quantify  this  infonnation.  Particularly  at  the  team  level,  quantification  makes  it  easier  to 
assess  the  accuracy  of  the  knowledge,  to  aggregate  individual  results  to  generate  a 
representation  of  team  knowledge,  or  to  compare  individual  results  within  a  team  to 
assess  knowledge  similarity.  In  fact,  the  relative  ease  with  which  certain  elicitation 
methods  are  quantified  provides  yet  another  reason  for  the  common  application  of 
conceptual  methods  (Blickensderfer,  et  al.,  1997b;  Gualtieri,  et  ah,  1996;  Kraiger,  et  ah, 
1996;  Mathieu,  et  al.,  in  press;  Stout,  1995;  Minionis,  et  al.,  1995;  Walsh,  et  al.,  1988)  or 
questionnaires  (Blickensderfer,  et  al.,  1997a;  1997b;  Jenkins  &  Rentsch,  1995)  to  the 
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team  arena.  However,  results  of  other  knowledge  elicitation  methods  such  as 
unstructured  interviews,  observations,  and  verbal  reports  can  also  be  quantified,  though 
often  with  more  effort.  For  instance,  verbal  responses  or  observed  events  can  be 
categorized  and  the  coded  data  can  be  quantified  in  terms  of  category  frequencies.  Even 
more  information  can  be  captured  by  examining  transition  frequencies  between 
categories.  Several  systematic  techniques  for  protocol  analysis  are  instructive  regarding 
these  procedures  (e.g.,  Cooke,  et  al.,  1996;  Ericsson  &  Simon,  1996;  vanSomeren,  et  al., 
1994). 

In  the  following  sections  we  review  some  metrics  that  can  be  derived  from  the 
results  of  knowledge  elicitation  methods  and  used  to  assess  team  knowledge.  Some  of 
these  metrics  have  been  commonly  used  and  others  are  relatively  novel.  Illustrations  of 
each  metric  are  provided  within  the  context  of  two  elicitation  examples.  In  both  cases, 
we  assume  that  the  team  is  an  aviation  team  with  two  team  members  (i.e.,  pilot  and 
navigator). 

The  first  example  of  elicitation  is  a  traditional  10-item,  four  alternative,  multiple 
choice  test  of  declarative  knowledge  which  lacks  the  open-endedness  of  a  typical 
knowledge  elicitation  technique,  and  in  fact,  would  be  better  described  as  an  assessment 
method,  but  illustrates  the  metrics  quite  well.  Hypothetical  responses  from  each  team 
member  are  presented  in  Table  2,  along  with  the  key  to  correct  responses.  The  second 
example  is  a  more  typical  conceptual  elicitation  method,  which  involves  the  collection  of 
pairwise  relatedness  ratings  of  5  task-relevant  terms  from  each  team  member,  and 
analysis  of  these  data  via  the  Pathfinder  network  scaling  program  (Schvane veldt,  1990). 
Hypothetical  rating  responses  and  network  results  are  presented  in  Figure  3.  For  both 
examples,  we  assume  that  the  knowledge  elicitation  is  perfect  in  that  the  domain 
knowledge  can  be  completely  captured  by  responses  to  the  10  items  or  10  pairwise 
ratings. 

[Insert  Table  2  and  Figure  3  about  here] 
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Similarity  metrics.  Research  on  team  knowledge  has  focused  most  often  on  the 
similarity  of  knowledge  within  a  team,  with  "similarity"  also  referred  to  as  consensus, 
convergence,  agreement,  compatibility  or  overlap  among  team  members.  In  addition, 
there  has  been  a  general  implication  that  high  intrateam  similarity  should  lead  to  the  most 
effective  teams  (Blickensderfer,  et  ah,  1997b;  Converse,  et  al,  1991;  Gualtieri,  Burns, 
Phipps,  Reeves,  &  Pierce,  1998;  Rentsch  &  Hall,  1994;  Stout,  1995).  However,  this 
assumption  emphasizes  the  "hold  in  common"  sense  of  knowledge  sharing  described 
previously,  and  undermines  the  "apportion"  perspective  which  is  most  appropriate  for 
heterogeneous  teams  in  which  team  members  have  distinct  roles  and  bring  different 
views  to  the  table.  In  heterogeneous  teams  at  least  some  knowledge  should  be 
associated  uniquely  with  distinct  team  roles,  and  for  this  knowledge  intrateam  knowledge 
similarity  would  be  low.  On  the  other  hand,  even  for  the  most  heterogeneous  teams, 
some  knowledge  would  likely  be  held  in  common  and  thus  knowledge  similarity  would 
be  a  useful,  albeit  insufficient,  descriptor  of  team  knowledge. 

How  can  similarity  be  measured?  In  the  test  example  in  Table  2  similarity  can  be 
measured  in  terms  of  the  number  or  percentage  of  responses  that  are  identical  for  two 
team  members  (6  or  60%  in  this  case).  In  general,  agreement  is  relatively  easy  to 
measure  in  cases  in  which  there  is  a  fixed  set  of  potential  responses  and  all  team 
members  respond  to  each  query,  as  on  a  test  or  questionnaire.  To  leverage  these 
advantages  of  tests,  some  researchers  have  taken  an  indirect  approach  and  addressed 
team  mental  models  in  tenns  of  shared  expectations  with  the  assumption  that  if  the 
mental  models  are  compatible  then  the  expectations  derived  from  them  should  be 
common  (Blickensderfer,  et  al.,  1997a;  1997b;  Cannon-Bowers,  et  al.,  1993). 
Expectations  can  be  measured  by  questionnaires  and  overlap  between  questionnaire 
response  can  be  measured  via  correlations  or  percent  identical  responses  depending  on 
the  nature  of  the  responses.  Furthermore,  intrateam  similarity  can  also  be  thought  of  as  a 
measure  of  interrater  reliability  at  the  group  level  (James,  Demaree,  &  Wolf,  1984). 
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Output  from  conceptual  methods  can  also  be  compared  (Shaw  &  Gaines,  1988). 

For  instance,  the  Pathfinder  procedure  is  associated  with  a  similarity  measure 
(Goldsmith  &  Davenport,  1990)  that  is  based  on  the  proportion  of  shared  links  in  two 
networks.  In  the  example  in  Figure  3,  the  two  resulting  networks  have  2  links  in 
common  (i.e.,  route-weather  and  instr.  check-take  off)  out  of  a  total  of  9  unique  links 
making  the  proportion  of  common  links  equal  to  .22.  It  turns  out  that  there  is  a  high 
probability  (.30)  of  obtaining  this  similarity  value  by  chance,  and  therefore,  one  would 
conclude  that  there  is  little  evidence  that  the  two  networks  are  similar.  A  variety  of 
investigators  have  used  this  metric  to  determine  intrateam  similarity  (Blickensderfer,  et 
ah,  1997b;  Cooke,  Stout,  Rivera,  &  Salas,  1998;  Gualtieri,  et  ah,  1996;  1998;  Kraiger,  et 
ah,  1996;  Stout,  1995).  Other  conceptual  methods  are  associated  with  parallel  means  of 
determining  similarity  such  as  comparisons  of  concept  centralities  in  UCINETs  (Mathieu 
et.  ah,  in  press)  and  Euclidean  distance  between  individuals  or  differences  in  dimension 
weights  in  multidimensional  scaling  solutions  (Walsh,  et  ah,  1988).  Another  alternative 
is  to  simply  correlate  the  original  pairwise  ratings  for  each  pair  of  team  members  (e.g., 
Blickensderfer,  et  al.  1997b).  The  Pearson  correlation  for  the  two  sets  of  relatedness 
ratings  in  Figure  3  is  -.22.  Finally,  to  generate  a  team  index  of  intrateam  similarity  the 
similarities  for  all  pairs  of  team  members  would  be  aggregated  in  one  of  several  ways 
described  later. 

The  cases  in  which  intrateam  knowledge  similarity  has  been  measured  have 
generally  found  reliable  correlations  between  the  similarity  metric  and  measures  of  team 
performance  or  team  process;  high  intrateam  knowledge  similarity  being  associated  with 
the  most  effective  teams  (Heffner,  Mathieu,  &  Cannon-Bowers,  1998;  Mathieu,  et  al.,  in 
press;  Minionis,  et  al.,  1995;  Stout,  1995;  Walsh,  et  al.,  1988).  Interestingly,  Mathieu 
et  al.,  (in  press)  found  this  relationship  to  hold  for  teamwork,  but  not  for  taskwork  and 
Minionis,  et  al.  (1995)  found  that  it  held  for  tasks  requiring  interdependence  and 
integrated  team  actions.  This  relationship  between  knowledge  similarity  and  team 
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performance  should  only  be  expected  for  homogeneous  teams  or  teams  that  are 
homogeneous  with  respect  to  the  knowledge  that  is  tested.  However,  even  for  a 
homogeneous  team,  knowledge  similarity  is  an  insufficient  metric  without  some  measure 
of  knowledge  accuracy. 

Accuracy  metrics.  Whether  one  argues  that  teams  should  have  similar  or  dissimilar 
knowledge,  it  is  clear  that  accuracy  is  crucial.  That  is,  all  team  members  could  have 
similar  knowledge  and  they  could  all  be  dead  wrong.  Alternatively,  in  a  heterogeneous 
team,  all  members  could  have  dissimilar  knowledge,  yet  overall,  the  team  could  be  highly 
accurate.  Indeed,  Mathieu  et  al.  (in  press)  found  that  accuracy  and  similarity  interact  to 
affect  team  performance. 

How  should  accuracy  be  measured?  In  the  test  example  it  is  simply  the  number  or 
percentage  correct  for  each  individual,  aggregated  across  the  team.  So  in  our  example 
the  pilot  has  4  or  40%  correct  and  the  navigator  has  6  or  60%  correct.  We  could  average 
their  scores  to  get  a  team  accuracy  score  of  50%.  Note  that  in  this  example  the  two  team 
members  could  have  100%  similarity  and  0%  accuracy  if  they  both  responded  in  the 
same  incorrect  way  to  each  response.  Also,  in  this  example,  if  team  members  respond 
differently  to  each  item,  the  upper  bound  on  the  average  team  accuracy  is  50%  because 
half  of  the  responses  by  definition  will  be  incorrect.  Accuracy  could  be  higher  however, 
in  a  case  in  which  there  were  multiple  "correct"  responses. 

To  detennine  accuracy  in  the  Pathfinder  example,  the  network  of  each  team 
member  needs  to  be  compared  to  a  referent.  The  referent  network  could  either  be 
constructed  logically  by  domain  experts  or  empirically,  by  collecting  ratings  from 
domain  experts.  In  addition,  similarity  between  the  team  member's  model  and  the 
referent  model  can  be  computed  in  several  ways  as  described  in  the  previous  section. 

For  purposes  of  the  example  in  Figure  3,  we  assume  that  the  referent  network  that 
represents  a  team  member  who  had  complete  and  accurate  knowledge  of  both  tasks 
would  contain  all  nine  of  the  unique  links  contained  in  the  two  networks  in  Figure  3. 
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Thus,  the  pilot  has  four  of  the  nine  referent  links  (44%)  and  the  navigator  has  seven  of 
them  (78%).  On  average  the  team’s  knowledge  is  61%  accurate. 

The  few  investigators  who  have  looked  at  accuracy  of  team  knowledge  have  done 
so  in  ways  similar  to  the  referent  approach  described  above.  Kraiger,  et  al.  (1996) 
compared  individual  networks  to  those  of  a  referent  and  averaged  this  value  across  team 
members  to  get  team  accuracy.  They  found  that  this  team  metric,  which  they  labeled 
"complementary  scoring  strategy"  was  correlated  with  scores  on  a  test  of  structural 
knowledge.  Mathieu,  et  al.  (in  press)  associated  each  team  member's  knowledge  results 
with  an  expert  of  a  particular  ranking  by  using  discriminant  analysis.  Team  knowledge 
accuracy  was  found  to  be  related  to  team  process  and  team  performance. 

Although  these  accuracy  metrics  are  relatively  straightforward,  they,  like  similarity 
metrics  tend  to  assume  that  the  team  in  question  is  homogeneous  with  respect  to 
knowledge.  The  knowledge  of  each  individual  team  member  is  compared  to  a  referent 
which  represents  that  knowledge  pertinent  to  the  team  task.  Thus,  heterogeneous  teams 
with  individuals  who  specialize  in  a  portion  of  the  knowledge  base  would  score  poorly 
using  this  kind  of  global  accuracy  metric.  However,  it  is  precisely  because  an  individual 
cannot  master  all  of  the  task  infonnation  that  a  team  is  required.  Thus,  what  is  needed  is 
an  accuracy  metric  for  heterogeneous  teams. 

Heterogeneous  accuracy  metrics.  According  to  the  definition  of  teams  offered 
earlier,  team  members  are  each  given  specific  roles  or  functions  to  perform.  Further,  if 
we  assume  that  this  division  of  labor  also  corresponds  to  specific  portions  of  the 
knowledge  base,  then  mastery  of  this  role-associated  knowledge  by  each  team  member 
should  be  indicative  of  team  knowledge  for  heterogeneous  teams. 

Although  there  has  been  little  research  in  which  heterogeneous  accuracy  has  been 
measured,  there  are  data  that  support  the  relation  between  heterogeneous  accuracy  and 
team  perfonnance.  Gualtieri,  et  al.  (1996),  for  instance,  measured  knowledge  similarity 
among  individuals  with  the  same  role,  but  on  different  teams,  and  found  that  within-role 
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similarity  was  greater  than  within-team  similarity,  supporting  the  need  for  heterogeneous 
knowledge  measures.  In  addition,  there  is  supporting  evidence  for  the  relation  between 
role  or  interpositional  knowledge  and  team  performance  when  the  former  is  manipulated 
through  training  interventions  (Volpe,  Cannon-Bowers,  Salas,  &  Spector,  1996;  Cannon- 
Bowers,  Salas,  Blickensderfer,  &  Bowers,  1998;  Minionis,  et  ah,  1995). 

How  should  heterogeneous  accuracy  be  measured?  The  basic  requirement  for  such 
a  measure  is  that  the  knowledge  base  be  chunked  or  partitioned  into  units  associated  with 
each  team  member  role.  These  chunks  could  consist  of  facts  or  concepts  that  are  either 
unique  to  each  role  or  that  are  partially  unique  in  that  some  concepts  overlap  roles.  For 
instance,  in  the  multiple  choice  test  example,  we  can  divide  the  knowledge  base  into 
unique  chunks  by  assuming  that  the  odd  numbered  questions  pertain  to  the  pilot  position 
and  the  even  numbered  questions,  to  the  navigator  position.  From  this  infonnation  we 
can  derive  the  percentage  of  role-relevant  items  responded  to  correctly  by  each  team 
member.  The  pilot  has  2/5  or  40%  of  the  pilot  items  correct  and  the  navigator  has  3/5  or 
60%  of  the  navigator  items  correct.  This  gives  us  an  average  of  50%  heterogeneous 
accuracy  for  this  team. 

Chunks  may  also  consist  of  the  same  material  or  concepts,  but  may  represent  a 
different  view  or  perspective  on  that  information.  For  example,  using  the  Pathfinder 
example,  different  perspectives  on  the  same  concepts  can  be  represented  as  role-specific 
referent  networks  in  which  the  concepts  are  identical,  but  the  links  between  concepts 
vary.  In  particular,  assume  that  the  successful  pilot  should  have  links  between  take  off 
and  instrument  check  ,  take  off  and  weather,  and  route  and  weather,  whereas  the 
successful  navigator  should  have  links  between  weather  and  take  off,  weather  and  reach 
altitude  and  route  and  instrument  check.  Thus,  out  of  the  five  unique  links  associated 
with  the  pilot  and  the  pilot-referent,  two  are  shared  (40%).  The  navigator  shares  3  out  of 
7  links  with  the  navigator  referent  (43%)  resulting  in  42%  team  heterogeneous  accuracy. 
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The  flip  side  of  heterogeneous  accuracy  is  accuracy  with  respect  to  the  knowledge 
associated  with  roles  other  than  your  own,  or  interpositional  accuracy.  Even  members  of 
heterogeneous  teams  may  be  versed  in  some  of  the  knowledge  pertinent  to  other 
positions.  This  is,  in  fact,  the  goal  of  cross  training— to  increase  the  team  members' 
understanding  of  the  task  from  the  others’  points  of  view  (Cannon-Bowers,  et  ah,  1998). 
The  degree  to  which  an  individual  has  mastered  role-oriented  information  over 
interpositional  information  provides  an  indication  of  degree  of  specialization.  A  team 
with  high  heterogeneous  accuracy  or  with  high  interpositional  accuracy  may  or  may  not 
be  specialized.  Specialization  depends  on  the  relative  mastery  of  the  position-specific 
material  over  the  interpositional  material. 

In  the  multiple  choice  example,  the  pilot  has  2/5  or  40%  of  the  navigator  items 
(even-numbered)  correct  and  the  navigator  has  3/5  or  60%  of  the  pilot  items  (odd- 
numbered)  correct.  Because  the  differences  between  these  scores  and  the  heterogeneous 
accuracy  scores  is  0  for  each  individual  and  thus,  0  for  the  team,  we  could  conclude  that 
this  particular  team  is  not  highly  specialized.  A  zero  (or  negative)  difference  indicates 
that  team  members  know  as  much  or  more  about  other  positions  as  they  do  of  their  own. 
Likewise,  the  similarity  between  each  team  member's  network  and  the  other  team 
member's  referent  can  also  be  calculated  (0  for  the  pilot  and  43%  for  the  navigator). 

Thus,  in  this  example  heterogeneous  accuracy  is  at  42%,  and  interpositional  knowledge 
at  22%.  The  difference  is  20%,  indicative  of  more  specialization  than  in  the  multiple 
choice  example. 

These  metrics  associated  with  heterogeneous  accuracy  have  received  preliminary 
support.  Cooke,  et  al.(  1998)  found  that  over  the  course  of  a  team  study,  teams  who  were 
largely  heterogeneous  and  specialized  after  training,  tended  to  increase  in  terms  of 
interpositional  accuracy  and  decrease  in  specialization.  The  degree  to  which  this  shift 
occurred  was  related  to  effective  team  perfonnance. 
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There  are  other  possibilities  for  heterogeneous  accuracy  metrics  as  well.  For 
instance,  Jenkins  and  Rentsch  (1995)  conceive  of  and  measure  "schema  accuracy"  in 
terms  of  the  team  member's  ability  to  describe  another  member's  schema— accurate 
knowledge  about  another's  knowledge,  much  like  the  measure  of  interpositional 
knowledge  described  above.  The  metric  is  based  on  the  sum  of  the  absolute  difference 
between  ratings  of  importance  made  by  a  team  member  about  another  team  member  and 
the  same  ratings  made  by  that  team  member  in  question.  Jenkins  and  Rentsch  (1995) 
label  the  combination  of  this  metric  and  an  agreement  metric,  "coorientation."  They 
found  that  the  interpositional  metric  predicted  team  effectiveness  to  a  greater  extent  than 
intrateam  similarity,  supporting  the  general  importance  of  heterogeneous  metrics. 

Heterogeneous  accuracy  metrics  provide  a  means  of  evaluating  the  degree  to  which 
individual  team  members  have  mastered  portions  of  the  knowledge  base,  and  hence  focus 
on  how  each  team  member's  knowledge  maps  onto  portions  of  the  knowledge  base, 
rather  than  how  portions  of  the  knowledge  base  map  onto  individual  team  members. 

So,  these  measures  do  not  indicate  the  extent  to  which  a  particular  concept  or  fact  has 
been  mastered  by  the  team.  Unless  heterogeneous  accuracy  is  perfect  (i.e.,  100%),  there 
is  no  guarantee  that  each  concept  has  been  mastered  by  at  least  one  team  member.  That 
is,  the  heterogeneous  accuracy  metrics  do  not  reveal  gaps  in  the  knowledge  base.  In 
addition,  although  teams  with  high  interpositional  accuracy  will  tend  to  have  knowledge 
that  is  more  distributed  across  team  members  than  a  team  with  low  interpositional 
accuracy,  the  degree  to  which  each  concept  or  fact  is  distributed  is  not  revealed.  These 
issues  can  be  resolved,  however,  with  knowledge  distribution  metrics. 

Knowledge  distribution  metrics.  Team  knowledge,  like  individual  knowledge  may 
be  associated  with  gaps  or  missing  information.  One  benefit  of  a  team  is  that  many  gaps 
in  individual  knowledge  can  be  compensated  for  by  the  knowledge  of  other  team 
members.  However,  gaps  in  team  knowledge  may  be  particularly  critical  such  as  when 
all  team  members  of  a  flight  crew  know  how  to  take-off,  but  no  one  knows  how  to  land. 
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In  general,  the  manner  in  which  specific  knowledge  is  distributed  among  members  of  a 
team  may  be  a  critical  factor  in  team  performance  (Hutchins,  1995). 

Along  these  lines,  Walsh,  et  al.  (1988)  distinguish  agreement  among  belief 
structures  from  information  coverage.  Walsh  et  al.  (1988)  measure  coverage  in  terms  of 
individual  weights  on  dimensions  in  multidimensional  scaling  solutions.  If  all 
dimensions  are  highly  weighted  by  at  least  one  team  member  then  coverage  is  good. 

They  found  that  coverage  predicted  team  performance,  but  that  surprisingly,  low 
coverage  was  associated  with  better  performance  than  high  coverage.  Note  that  the 
concept  of  coverage  necessitates  some  standard  against  which  to  judge  completeness  (in 
this  case  the  set  of  MDS  dimensions). 

How  should  knowledge  distribution  be  measured?  In  the  test  example,  in  which  it 
is  assumed  that  the  domain  is  completely  covered  by  the  set  of  10  items,  items  2,  3,  and 
7-10  were  answered  correctly  by  at  least  one  team  member,  so  we  could  say  that  the 
team  has  mastered  60%  of  the  knowledge  base.  Further,  four  of  these  six  mastered  items 
were  answered  correctly  by  both  team  members,  so  we  could  say  that  40%  of  the 
knowledge  is  redundantly  distributed  across  team  members,  20%  is  uniquely  distributed, 
and  40%  is  not  covered.  This  same  kind  of  analysis  could  be  done  considering  role- 
associated  portions  of  the  knowledge  base,  rather  than  individual  items.  In  the  test 
example,  a  criterion  of  60%  might  be  established  to  indicate  mastery  of  either  the  pilot  or 
navigator  role.  If  this  were  the  case,  there  would  be  mastery  of  both  roles  with  the 
navigator  alone  mastering  each  so  that  knowledge  distribution  would  be  considered 
unique. 

The  Pathfinder-based  metric  for  knowledge  distribution  is  similar.  At  the  role  level 
for  instance,  mastery  can  be  defined  as  having  at  least  60%  network  similarity  with  the 
role  referent.  In  the  example  in  Figure  3,  no  individual  has  mastered  either  role,  so  that 
there  is  0  coverage  for  this  information.  In  addition,  mastery  of  a  single  concept  can  be 
defined  in  network  terms  as  the  proportion  of  shared  links  associated  with  that  concept 
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for  a  given  network  and  a  referent.  As  in  the  illustration  of  accuracy,  the  referent 
network  is  assumed  to  be  the  nine  links  in  the  union  of  links  in  the  two  networks  of 
Figure  3.  Thus,  the  mastery  of  the  concept  route  is  based  on  the  presence  of  the  three 
links  connecting  route  to  each  of  reach  altitude,  instrument  check,  and  weather.  The 
pilot  has  one  of  these  links  and  the  navigator  has  all  of  them  resulting  in  33%  and  100% 
respective  concept  mastery  scores.  Concept  mastery  can  be  computed  for  each  of  the 
other  concepts  in  the  same  way.  In  this  example,  the  pilot  has  25%  mastery  of  reach 
altitude  ,  66%  mastery  of  take  off,  and  50%  mastery  of  the  remaining  items,  whereas  the 
navigator  has  66%  mastery  of  take  off  and  75%  mastery  of  the  remaining  items.  A  cutoff 
value  for  proportion  shared  links  can  be  used  to  classify  concepts  as  mastered  or 
unmastered  and  to  ultimately  identify  the  degree  to  which  conceptual  knowledge 
distributed  among  team  members.  In  this  example,  if  at  least  60%  mastery  is  required, 
then  the  team,  in  this  case  the  navigator,  has  complete  coverage  of  the  live  items, 
however  the  knowledge  is  uniquely  distributed  among  team  members. 

Mapping  team  metrics  to  team  knowledge.  The  team  metrics  described  above  can 
be  applied  to  results  from  any  of  the  elicitation  methods.  There  are  some  preconditions 
however,  including  the  ability  to  assess  similarity  between  the  two  elicitation  outcomes, 
and  in  the  case  of  accuracy  and  distribution  metrics,  some  assumptions  about  the 
knowledge  required  of  the  team  or  of  individual  roles.  Although  the  team  metrics  are  not 
specific  to  type  of  knowledge  or  rate  of  change  of  that  knowledge,  the  choice  of  a  metric 
does  depend  on  assumptions  regarding  the  distribution  of  that  knowledge  among  team 
members.  Under  assumptions  of  homogeneous  distribution  in  which  all  team  members 
possess  more  or  less  the  same  knowledge,  similarity  and  accuracy  metrics  are 
appropriate.  Under  assumptions  of  heterogeneous  distribution,  however,  the 
heterogenerous  accuracy  and  knowledge  distribution  metrics  are  relevant.  In  the 
following  section  we  describe  methods  for  aggregating  the  metric  values  obtained  for 
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individual  team  members  (accuracy,  heterogenerous  accuracy)  or  pairs  of  team  members 
(similarity)  in  order  to  generate  a  single  value  representative  of  the  team  as  a  whole. 

Aggregation  Methods 

Given  a  metric  derived  from  a  collective  measurement  approach  (i.e.,  administered 
to  individual  team  members),  how  should  this  metric  be  combined  across  team  members 
to  produce  a  metric  representative  of  the  team  as  a  whole?  There  are  many  ways  to 
aggregate,  although  most  researchers  have  averaged  the  individual  data  (Blickensderfer, 
et  ah,  1997a;  1997b;  Jenkins  &  Rentsch,  1995;  Kraiger,  et  ah,  1996).  However,  the 
danger  in  averaging  is  that,  due  to  team  member  variance,  the  averaged  result  may  be 
unrepresentative  of  any  single  team  member.  Alternatively,  some  investigators  have 
approached  aggregation  by  using  the  median  value  (Hinsz,  1995)  or  by  relying  on 
responses  given  by  the  majority  (Minionis,  et  ah,  1995)  or  all  (Gualtieri  et  ah,  1996)  of 
the  team  members.  The  two  generic  aggregation  methods  represented  in  Figure  1 
represent  any  two  of  the  many  possibilities. 

There  are  other  possibilities  too,  such  as  taking  the  sum,  the  minimum  or 
maximum,  value,  or  the  range.  The  minimum  and  maximum  strategies  would  represent 
team  knowledge  to  the  extent  that  knowledge  was  a  function  of  the  strongest  or  weakest 
team  member.  For  instance,  one  member's  knowledge  may  predict  team  performance  on 
an  aspect  of  the  task  that  relies  heavily  on  that  individual  (e.g.,  piloting  a  plane). 
Blickensderfer  et  ah,  (1997b)  found  that  the  best  predictor  of  team  performance  was 
when  the  aggregate  for  a  shared  representation  survey  was  based  on  the  minimum 
intrateam  pairwise  correlation  for  responses  on  that  survey.  The  range  may  be  a  good 
measure  of  team  knowledge  to  the  extent  that  intragroup  variation  on  some  measure  is 
predictive  of  team  knowledge  (e.g.,  teams  that  are  heterogeneous  on  some  knowledge 
measure  may  be  more  or  less  effective  than  teams  that  are  more  or  less  consistent). 
Finally,  Walsh,  et  al.  (1988)  distinguished  potential  team  knowledge  from  actual  team 
knowledge,  the  latter  being  based  on  an  aggregate  that  was  weighted  by  the  degree  to 
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which  team  members  participated  in  team  activities.  The  idea  here  was  that  the 
knowledge  of  the  more  vocal,  active  team  members  would  play  a  greater  role  in 
determining  the  knowledge  of  the  team  as  a  whole. 

Clearly  there  is  no  definitive  method  for  aggregating  individual  data  and  no  clear 
means  of  using  the  team  knowledge  features  to  select  an  aggregation  method.  Research 
is  needed  to  comparatively  evaluate  these  and  other  methods.  As  a  general  rule, 
however,  it  is  good  policy  to  avoid  averaging  team  member  data  that  vary  greatly  and  to 
instead  rely  on  an  alternative  approach  like  taking  the  minimum  or  maximum. 

One  final  note  regarding  aggregation  is  that  it  may  be  more  meaningful  to  use  the 
individual  metrics  to  discretely  categorize  a  team,  rather  than  to  characterize  it  by  an 
aggregate  score.  For  example,  Cooke,  et  al.,  (1998)  collected  pairwise  relatedness 
ratings  for  task-related  concepts  from  each  team  member.  A  measure  of  intrateam 
similarity  was  derived  by  summing  rating  correlations  or  network  similarity  measures  for 
each  pair  of  the  three  team  members.  Although  the  resulting  metric  was  not  generally 
predictive  of  performance,  a  different  picture  emerged  when  a  median  cutoff  was  used  to 
determine  pairwise  team  member  similarity.  This  information  was  used  to  generate  the 
patterns  displayed  in  Figure  4.  Interestingly,  Pattern  A  is  associated  with  the  most 
effective  team  and  Pattern  B  with  the  least  effective.  Further  exploration  of  this  kind  of 
discrete  representation  of  team  metrics  seems  warranted. 

[Insert  Figure  4  about  here] 

Conclusions 

Team  knowledge  is  central  to  a  number  of  theoretical  explanations  of  team 
performance  ranging  from  team  decision  making  to  team  situation  awareness.  There  has 
been  much  speculation  concerning  the  nature  of  team  knowledge  and  the  role  that  it  plays 
in  team  cognition  and  perfonnance.  In  addition,  recent  applications  in  team  training  and 
team  assessment  require  an  approach  that  goes  beyond  assessment  to  uncover  the  content 
and  structure  of  team  knowledge.  Only  recently  have  investigators  begun  to  explore 
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measures  that  might  capture  the  essence  of  team  knowledge.  The  majority  of  the 
research  on  this  topic  has  focused  on  the  measurement  of  the  more  stable  and  generic 
team  mental  models,  has  used  methods  of  interviews,  questionnaires  and  conceptual 
methods  to  elicit  this  knowledge,  and  has  derived  metrics  of  similarity  and  to  a  lesser 
extent,  accuracy  from  the  results  of  these  methods.  To  this  point  the  results  are 
promising,  with  these  measures  of  team  mental  models  corresponding  to  various  team 
process  and  perfonnance  measures. 

[Insert  Table  3  about  here] 

Despite  this  progress,  much  remains  to  be  done.  Table  3  lists  some  future  research 
directions  in  terms  of  methodological  requirements  for  the  measurement  of  team 
knowledge.  First,  current  methods  for  eliciting  team  knowledge  all  approach  elicitation 
from  the  level  of  the  individual.  Because  team  knowledge  is  thought  to  be  more  than  the 
sum  of  individual  team  members'  knowledge,  holistic  approaches  to  measurement  are 
needed  which  elicit  knowledge  from  the  team  as  a  whole.  Second,  team  knowledge 
elicitation  and  knowledge  elicitation  methodologies  in  general,  need  to  address  the  more 
fleeting,  context-specific  understanding  of  a  situation  that  we  call  team  situation  models. 
This  type  of  knowledge  is  thought  to  underlie  team  situation  awareness  (Cooke,  et  al., 
1997;  Stout,  et  al.,  1996)  and  other  team  behavior  in  complex  dynamic  tasks.  In  fact,  in 
tasks  in  which  the  situation  is  continually  and  rapidly  changing,  team  differences  in  this 
dynamic  understanding  may  account  for  the  most  variance  in  team  behavior.  As  a  first 
step,  we  have  suggested  some  ways  that  current  elicitation  techniques  could  be  adapted 
or  modified  for  this  job  that,  for  example,  rely  on  data  collected  unintrusively  during  the 
course  of  team  performance  (i.e.,  keystrokes,  communication  events).  Other  approaches 
should  be  considered  as  well. 

Additional  work  is  also  needed  on  metrics  that  take  the  output  of  the  knowledge 
elicitation  methods  and  produce  values  that  are  meaningful  in  respect  to  team  knowledge. 
There  are  a  number  of  interesting  metrics  that  can  be  derived  from  elicitation  data  that 


Team  Knowledge  -  34 


can  represent  team  knowledge  in  heterogeneous  teams.  The  heterogeneous  accuracy  and 
knowledge  distribution  metrics  only  scratch  the  surface.  In  general,  more  measurement 
work  is  needed  that  reflects  the  heterogeneity  assumed  to  be  central  to  the  accepted 
definition  of  teams. 

Further  exploration  of  ways  to  discretely  classify  teams  based  on  these  metrics  also 
appears  to  be  a  promising  direction.  Two  teams  may  have  the  same  average  intrateam 
similarity  value,  but  are  differentially  classified  when  the  pattern  by  which  similarity  is 
dispersed  is  taken  into  account.  The  pattern  may  better  reflect  team  knowledge. 

Furthennore,  the  methods  and  metrics  discussed  in  this  paper  follow  from  a  specific 
theoretical  view  of  team  knowledge,  but  there  are  other  views.  More  importantly,  there 
are  other  factors  that  are  not  addressed  by  this  view,  but  that  seem  to  be  relevant  to  team 
knowledge  such  as  attitudes  of  team  members  or  positions  of  authority  on  the  team.  In 
addition,  there  are  related  approaches  that  may  also  be  relevant  to  team  knowledge  such 
as  dynamic  systems  and  multiple  distributed  agents.  Additional  promising  methods  and 
metrics  may  arise  from  a  broader  view  of  team  knowledge. 

Finally,  despite  the  most  prolific  methodological  developments  in  this  area,  the 
methods  are  only  valuable  to  the  extent  that  they  have  been  validated  and  that  they  are 
associated  with  principles  for  selecting  appropriate  methods  in  a  given  situation.  The 
variety  of  metrics  described  in  this  paper  illustrate  the  richness  and  complexity  of  team 
knowledge  measurement.  The  innumerable  combinations  of  knowledge  elicitation 
method,  team  metric,  and  aggregation  method,  not  to  mention  the  future  possibilities  of 
holistic  and  continuous  elicitation  methods,  can  seem  overwhelming.  However,  given 
that  team  knowledge  is  equally  multifaceted,  such  options  seem  necessary.  Many 
methodological  decisions  can  be  guided  analytically  by  the  nature  of  the  team  knowledge 
that  is  to  be  measured  and  by  the  specific  features  of  the  measurement  methodology 
(e.g.,  use  conceptual  methods  or  interviews  or  questionnaires  for  measuring  team  mental 
models;  use  heterogeneous  accuracy  metrics  for  determining  the  accuracy  of  a  team  task 
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associated  with  distributed  roles  and  knowledge).  As  measurement  methods  are 
continually  refined  are  used  in  team  research,  additional  guidelines  for  their  usage  should 
be  revealed. 

Additionally,  empirical  validation  can  also  speak  to  the  selection  of  an  appropriate 
methodology.  Measures  of  team  knowledge  should  be  reliable.  They  should  also  be 
valid  in  that  they  differentiate  teams  that  are  assumed  to  differ  in  the  targeted  team 
knowledge,  either  as  a  result  of  experience  or  manipulation  of  the  training  or  task 
environment.  Further,  given  that  team  knowledge  is  relevant  to  team  perfonnance,  then 
effective  measures  of  team  knowledge  should  correspond  to  differences  in  team 
performance.  It  is  also  useful  to  compare  multiple  measures  of  team  knowledge  or 
variations  in  the  same  measure  against  these  criteria.  Limited  empirical  evaluations  of 
team  knowledge  measures  have  been  conducted.  For  instance,  knowledge  elicitation 
methods  have  been  comparatively  evaluated  in  experiments  at  the  level  of  individual 
elicitation,  but  more  work  is  needed  on  applications  of  these  methods  to  the  team  front. 
Also  as  previously  described,  empirical  evaluative  data  exists  for  similarity  and  accuracy 
metrics,  but  not  for  heterogenerous  metrics. 

It  is  likely  that  the  most  mileage  (in  a  sense  of  predicting  team  performance)  is  to  be 
gained  by  using  composite  measures.  For  instance,  it  may  be  that  some  combination  of 
knowledge  similarity,  knowledge  distribution,  and  heterogeneous  accuracy  is  associated 
with  optimal  team  performance  in  a  given  domain.  Cooke,  et  al.  (1998)  found  that  for  a 
relatedness  rating-based  measure,  both  overall  knowledge  accuracy  and  interpositional 
knowledge  accuracy  predicted  team  performance  in  a  simulated  helicopter  mission,  with 
better  teams  being  more  accurate  and  having  more  accurate  interpositional  knowledge. 
Additionally,  Minionis,  et  al.  (1995)  successfully  employed  a  single  measure  of  overlap 
that  took  both  agreement  and  accuracy  into  account.  Comparative  evaluations  should 
also  suggest  promising  suites  of  methods. 
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In  conclusion,  the  measurement  of  team  knowledge,  though  challenging,  stretches 
the  limits  of  existing  measurement  approaches  and  cognitive  theories.  Traditional 
theories  and  measures  have  been  developed,  applied,  and  evaluated  for  much  simpler 
tasks  in  single-operator  settings.  Investigations  of  team  knowledge  and  the  development 
of  appropriate  measures  of  team  knowledge  extends  previous  work  to  complex 
sociotechnical  environments.  This  rich  context  is,  after  all,  where  much  cognition  occurs 
and  where  many  applied  problems  begin. 
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Table  1.  A  framework  for  circumscribing  team  knowledge.  This  paper  focuses  on  that 
part  of  team  cognition  that  is  within  the  box. 


Team  Cognition 

Team  Decision  Making 
Team  Situation  Awareness... 


Team  Knowledge 

Team  Mental  Model 

•  long  lasting,  exists  prior  to  task 

•  acquired  through  training,  experience  etc. 

•  a  variety  of  content  (taskwork,  teamwork) 

•  a  variety  of  forms  (declarative,  procedural,  strategic) 

•  function:  collective  knowledge  base,  leads  to  common 

expectations 


world 


Team  Situation  Model 

•  fleeting,  dynamic 

•  acquired  during  task  using  team  mental  model  and 

cues 

•  situation-specific 

•  function:  interpret  situation  in  compatible  way 


...Team  Perception 
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Table  2.  Example  of  elicitation  of  factual  knowledge  for  two  team  members  (pilot  and 
navigator)  using  a  10-item  multiple  choice  test. 


Item  No. 

Correct  Responses 

Pilot's  Responses 

Navigator's  Responses 

1 

a 

c 

d 

2 

c 

a 

c 

3 

d 

b 

d 

4 

a 

b 

c 

5 

a 

d 

d 

6 

b 

c 

c 

7 

d 

d 

d 

8 

b 

b 

b 

9 

c 

c 

c 

10 

c 

c 

c 
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Table  3.  Methodological  needs  for  the  measurement  of  team  knowledge. 


1)  Development  of  holistic  team  knowledge  elicitation  methods. 

2)  Adaptation  and  development  of  methods  for  eliciting  team  situation  models 

including... 

a.  the  adaptation  of  process  tracing  and  observation  methods 

b.  the  use  of  team  behavioral  data  as  input  to  conceptual  methods  (e.g.,PRONET; 

Cooke,  et  al.,  1996) 

c.  stream-lining  of  off-line  elicitation  methods. 

3)  Further  development  of  metrics  that  reflect  team  heterogeneity. 

4)  Exploration  of  discrete  classifications  of  teams  based  on  metrics. 

5)  Develop  measures  that  reflect  an  even  broader  view  of  team  knowledge. 

6)  Further  develop  the  principles  for  mapping  a  specific  measurement  method  onto  team 

knowledge  characteristics. 

7)  Test  the  predictive  validity  of  various  elicitation  methods,  metrics,  and  aggregation 

methods  in  terms  of  their  ability  to  predict  team  process  behaviors  and  team 
performance. 

8)  Comparative  evaluations  of  various  elicitation  methods,  metrics,  and  aggregation 


methods  against  each  other. 
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Figure  Captions 

Figure  1.  Characteristics  of  targeted  team  knowledge  and  team  knowledge  measures. 
Figure  2.  Collective  and  holistic  approaches  to  the  measurement  of  team  knowledge 
Figure  3.  Example  relatedness  ratings  from  two  team  members  (pilot  and  navigator)  and 
the  associated  Pathfinder  networks. 

Figure  4.  Four  patterns  of  knowledge  similarity  among  team  member  pairs.  P=pilot, 
NO=navigation  officer,  IO=intelligence  officer.  Those  enclosed  in  the  same  circle  have 
similar  conceptual  structures  based  on  a  median  cutoff.  Highest  performing  team  is  team 
9,  lowest  performing  team  is  team  8.  Team  2  was  ranked  second  after  team  9  on  two  of 
the  four  performance  measures 
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TARGETED  TEAM  KNOWLEDGE 


process  tracing 

conceptual 

methods 
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KNOWLEDGE  OF  INDIVIDUAL 
TEAM  MEMBERS 


i 


N 


Target  of  the 
Collective  Approach 


Target  of  the 
Holistic  Approach 


TEAM  BEHAVIOR 
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Pilot's  relatedness  ratings  j(l=related,  5=unrelated): 


route 

weather 

inst.  check 

take  off 

reach  alt. 

route 

0 

- 

- 

- 

- 

weather 

1 

0 

- 

- 

- 

instrument  check 

2 

1 

0 

- 

- 

take  off 

3 

2 

1 

0 

- 

reach  altitude 

4 

3 

2 

1 

0 

Navigator's  relatedness  ratings  (l=related,  5=unrelated): 


route 

route 

0 

weather 

inst.  check 

take  off 

reach  alt. 

weather 

1 

0 

- 

- 

- 

instrument  check 

1 

3 

0 

- 

- 

take  off 

3 

1 

1 

0 

- 

reach  altitude 

1 

1 

1 

3 

0 

Pilot's  Pathfinder  Network 


Navigator's  Pathfinder  Network: 
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